Database development is a relatively new technology dating back to around the 1960's. Initially, individual data items were grouped into database tables based upon some functional need. In the early 1970's, a data flow diagram method was developed to depict the data as inputs and outputs to data processing functions. The data groupings depicted as inputs and as outputs became database tables or data files. The data processing functions were programmed as computer software to read and write data into these database tables or data files. Around this same time, data normalization rules were formulated that dictated how to develop normalized groupings of data items where the normalized groupings were independent of the data processes that use them. Each so developed data item normalized grouping often became a database table. The data normalization rules were developed to create databases with minimal data storage requirements. At that time, data storage was a very expensive commodity and minimizing a database's data storage yielded a less expensive data system solution.
In 1976, the concept of entity-relationship diagramming and data modeling was developed. One function of data models is to design the structures of databases such as database tables and database table columns. A data model is a combination of the entity-relationship diagram and its associated metadata, all of which is typically stored in a computer memory.
Each data item or data attribute of a data model must also have metadata defined for it. Metadata is often referred to as the data about data. Metadata is information about specific data items such as the definition of the data item along with the type of data, the length and precision of the data, valid acceptable data values, and whether a data value is required or optional. The metadata of a data system is often recorded in a data model. Sometimes, the metadata of a data system, including the data model, is stored in a metadata repository. Over time, the entity-relationship diagram method replaced the data flow diagram method. Today, entity-relationship diagrams and data model concepts are used by the majority of database designers. Entity-relationship diagrams have become the basis of the process of data modeling.
In the late 1980's, another method to the data normalization rules based database table development was developed, which was referred to as “data warehouse design methodology”. It was, developed mainly for optimizing report generation speed from a database. Now, instead of the data storage efficiency provided via normalized database tables, the data warehouse database uses data redundancy to improve data retrieval speed. In the data warehouse database, redundant data abounds, which often improves the data retrieval speeds. The data warehouse development method begins with the same data normalization rules methods employed previously. However, the data items are then de-normalized in computer memory to form data warehouse database tables of two types. These two data warehouse database table types are the dimensional database tables and the fact database tables. In general, the fact type database tables contain mostly quantitative data items or data items that have numerical data values while the dimensional type database tables contain mostly non-quantitative data items or data items that have textual data values.
Databases developed and stored as data warehouses in a computer memory are additional to databases developed and stored in a computer memory using data normalization rules. In fact, data normalization developed databases often referred to as on-line transactional processing (OLTP) databases are the source of data for most data warehouse type databases. OLTP databases are used to collect data needed for the day to day operations of the organization while data warehouse databases use this collected data to generate complex reports about how the organization is performing. In some cases, data warehouse databases are sourced from unstructured data files usually associated with legacy mainframe data systems. These legacy mainframe data systems were often implemented before the data normalization rules were developed.
Alternative data model development methods have since been developed that are variations of the entity-relationship diagrams. These alternative data model development methods are also computer based software applications that include the Unified Modeling Language method and the Object-Oriented Data Modeling method. Also, some vendors supply skeletal data models stored in computer memory that are incomplete data models usually specific to certain industries. The purchased skeletal data models are then completed in computer memory where more specific data requirements are implemented.